37 research outputs found
Cramer Rao-Type Bounds for Sparse Bayesian Learning
In this paper, we derive Hybrid, Bayesian and Marginalized Cram\'{e}r-Rao
lower bounds (HCRB, BCRB and MCRB) for the single and multiple measurement
vector Sparse Bayesian Learning (SBL) problem of estimating compressible
vectors and their prior distribution parameters. We assume the unknown vector
to be drawn from a compressible Student-t prior distribution. We derive CRBs
that encompass the deterministic or random nature of the unknown parameters of
the prior distribution and the regression noise variance. We extend the MCRB to
the case where the compressible vector is distributed according to a general
compressible prior distribution, of which the generalized Pareto distribution
is a special case. We use the derived bounds to uncover the relationship
between the compressibility and Mean Square Error (MSE) in the estimates.
Further, we illustrate the tightness and utility of the bounds through
simulations, by comparing them with the MSE performance of two popular
SBL-based estimators. It is found that the MCRB is generally the tightest among
the bounds derived and that the MSE performance of the Expectation-Maximization
(EM) algorithm coincides with the MCRB for the compressible vector. Through
simulations, we demonstrate the dependence of the MSE performance of SBL based
estimators on the compressibility of the vector for several values of the
number of observations and at different signal powers.Comment: Accepted for publication in the IEEE Transactions on Signal
Processing, 11 pages, 10 figure
Variational Student: Learning Compact and Sparser Networks in Knowledge Distillation Framework
The holy grail in deep neural network research is porting the memory- and
computation-intensive network models on embedded platforms with a minimal
compromise in model accuracy. To this end, we propose a novel approach, termed
as Variational Student, where we reap the benefits of compressibility of the
knowledge distillation (KD) framework, and sparsity inducing abilities of
variational inference (VI) techniques. Essentially, we build a sparse student
network, whose sparsity is induced by the variational parameters found via
optimizing a loss function based on VI, leveraging the knowledge learnt by an
accurate but complex pre-trained teacher network. Further, for sparsity
enhancement, we also employ a Block Sparse Regularizer on a concatenated tensor
of teacher and student network weights. We demonstrate that the marriage of KD
and the VI techniques inherits compression properties from the KD framework,
and enhances levels of sparsity from the VI approach, with minimal compromise
in the model accuracy. We benchmark our results on LeNet MLP and VGGNet (CNN)
and illustrate a memory footprint reduction of 64x and 213x on these MLP and
CNN variants, respectively, without a need to retrain the teacher network.
Furthermore, in the low data regime, we observed that our method outperforms
state-of-the-art Bayesian techniques in terms of accuracy
Over-The-Air Clustered Wireless Federated Learning
Privacy, security, and bandwidth constraints have led to federated learning
(FL) in wireless systems, where training a machine learning (ML) model is
accomplished collaboratively without sharing raw data. Often, such
collaborative FL strategies necessitate model aggregation at a server. On the
other hand, decentralized FL necessitates that participating clients reach a
consensus ML model by exchanging parameter updates. In this work, we propose
the over-the-air clustered wireless FL (CWFL) strategy, which eliminates the
need for a strong central server and yet achieves an accuracy similar to the
server-based strategy while using fewer channel uses as compared to
decentralized FL. We theoretically show that the convergence rate of CWFL per
cluster is O(1/T) while mitigating the impact of noise. Using the MNIST and
CIFAR datasets, we demonstrate the accuracy performance of CWFL for the
different number of clusters across communication rounds.Comment: Under review at ICASSP 202
Seeing is Believing: A Federated Learning Based Prototype to Detect Wireless Injection Attacks
Reactive injection attacks are a class of security threats in wireless
networks wherein adversaries opportunistically inject spoofing packets in the
frequency band of a client thereby forcing the base-station to deploy
impersonation-detection methods. Towards circumventing such threats, we
implement secret-key based physical-layer signalling methods at the clients
which allow the base-stations to deploy machine learning (ML) models on their
in-phase and quadrature samples at the baseband for attack detection. Using
Adalm Pluto based software defined radios to implement the secret-key based
signalling methods, we show that robust ML models can be designed at the
base-stations. However, we also point out that, in practice, insufficient
availability of training datasets at the base-stations can make these methods
ineffective. Thus, we use a federated learning framework in the backhaul
network, wherein a group of base-stations that need to protect their clients
against reactive injection threats collaborate to refine their ML models by
ensuring privacy on their datasets. Using a network of XBee devices to
implement the backhaul network, experimental results on our federated learning
setup shows significant enhancements in the detection accuracy, thus presenting
wireless security as an excellent use-case for federated learning in 6G
networks and beyond.Comment: 6 pages with 8 figure